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rnPvmOHT NOTICE 

[00011 A portion of the disclosure of this patent document contains material which is subject to 

copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the 
patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or 
records, but otherwise reserves all copyright rights whatsoever. 

rn r>£S-R F.FERENCE TO RELATED APPLICATIONS 
[0002] The following applications are cross-referenced and incorporated herein by reference: 

[0003] U.S. Patent Application No. 10/125,346 [Attorney Docket No. FX/A1015], entitled 

SYSTEMS AND METHODS FOR DISPLAYING TEXT RECOMMENDATIONS DURING 
COLLABORATIVE NOTE TAKING, by Laurent Denoue, et aL, filed April 19, 2002. 
[0004] u.S.PatentApplicanonNo.lO/617,549[AttomeyDocketNo.FXPL-01062USOMCF/MLR; 

FX/A2008] , entitled TELEPRESENCE SYSTEM AND METHOD FOR VIDEO TELECONFERENCING, 
by Jonathan T. Foote, et al„ filed July 11, 2003. 
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[0005] U.S.PatentApplicationNo.l0/629,403[AttorneyDo C ketNo.FXPL-01063US0MCF/DJB; 
FX/A20 1 4], entitled A VIDEO ENABLED TELE-PRESENCE CONTROL HOST, by Qiong Liu, et al., filed 
July 28, 2003. 

irnrT n of THF DISCLOSURE 
[0006] Thepresentdisclosurerelatesgenem^ 

of digital content and more specifically to systems and methods for monitoring past and current user 
interaction with representations of digital content to improve the presentation of and interaction with 
representations of digital content. 

BACKGROUND 

[0007] Digital content is steadily becoming more ubiquitous in today's modem computing 

environment. Consequently, users are more consistently and frequently accessing digital content. As users 

accessdigitalcontentmoreandmo 

and interacted with is becoming more important. 

[0008] inmanyinstances,usersmayfindthemselvesusingandinteractingwithrepresentationsof 
digital contentthataresimilarbutdifferenttootherrepresentationsof digital content. For example, many 
users access digital content, such as software applications, web pages, documents, etc. at work using a 
desktop computing device and at home using a laptop, PDA, and/or cellphone. In a simple example, a user 
may interact with a representation of digital content that is merely of a different size than a previously used 
representation ofthe digital content. Consequently, users may interact with different representations ofthe 

same digital content. 
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[0009] A user may get confused, disoriented, or otherwise have trouble when interacting with a 

representation of digital content that is somehow different than a previous representation of the same content 
with which she interacted. For example, a user may have trouble selecting and interacting with 
representations that have been scaled for presentation witha PDA. Pixel coordinates maybe scaled-down 
such that selectable areas of the representation become harder to select. 

[0010] In many instances, the problem can be amplified by other factors such as the use of a stylus 

or fingers on a touch-screen. Scaled representations are often accessed or interacted with using non- 
traditional or less accurate input devices such as styluses or touch-screens. Scaled representations can 
further be more difficult to understand, especially when the representation includes text-based regions. 
Scaled-down representations such as those on touch screens can be more difficult to interact with because 
a user does not know what can be selected in many cases. Unlike a mouse or similar input device, touch 
screens typically do not have cursors that can change shape or otherwise provide feedback to indicate a 
selectable area. 

[001 1] Users may also find themselves accessing content that is similar to other content, yet not 

exactly the same. For example, users may access different web pages having similar layouts or features, 
such as text boxes including search fields, submit buttons, and portlets such as news headline portlets. 
Moreover, users may begin use of new software applications having interfaces with a similar layout or 
functionality to other interfaces. 

[0012] Users may get confused or disoriented when interacting with a representation of new digital 

content, such as an interface of a new software application or a new digital image. Users may have trouble 
determining where selectable features such as menu items are located. 
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RRTFF DESCRIPTION OF THF DRAWINGS 
[0013] Figure 1 is a flowchart illustrating a method in accordance with an embodiment for 

deforming a representation of digital content using past and current user interaction. 
[0014] Figure 2 is a system in accordance with an embodiment that can be used to implement the 

flowchart of Figure 1. 

[0015] Figure 3 is a flowchart in accordance with one embodiment of the present invention for 

deforming representations of digital content. 

[0016] Figure 4 is a system in accordance with an embodiment that can be used in the deformation 

of representations of digital content. 

[0017] Figure 5a illustrates an exemplary first representation of content, wherein interaction areas 

of the first representation have been determined. 

[0018] Figure 5b illustrates an exemplary second representation of the content of Figure 5a, 

wherein active area deformation has been applied to the second representation in accordance with an 
embodiment. 

[0019] Figure6aillustratesanexemplaryfirstrepresentationofdigitalcontenthavingdetermined 
interaction areas in accordance with an embodiment. 

[0020] Figures 6b - 6d illustrate an exemplary second representation of the digital content of 

Figure 6ahavingananimationapplied to corresponding interaction areas in accordance with an embodiment. 
[0021] Figure 7a is an exemplary first representation of a web browser interface having a first 

representation ofa web page presented through the webbrowser in accordance with an embodiment of the 
present invention. 



Attorney Docket No.: FX/A2005 
dcromer/fxpl/1 061 US0/1 061 usO.001 .app.wpd 



-5- 



[0022] Figure S 7b-7ci S anexemplaiysecondrepresentationofthewebbrowserinterfacehaving 
asecondrepresentationo^ 

been applied to the second representation of the web page in accordance with an embodiment of the present 
invention. 

[0023] Figure 8 is a system in accordance with an embodiment for using interaction information 

in a meeting software application. 

[0024] F igure9isasysteminaccordancewithanembodimentforaccessingcontentwitharemote 
device, wherein a representation of the content can be deformed for presentation on the remote device. 
[0025] Figure 10 is a flowchart in accordance with an embodiment for deforming representations 

of content on a remote device. 

[00261 Figure llaisaflowchartmaccoroto^^ 

is used at a remote device to deform representations of digital content. 

[0027] Figurellbisaflowchartinaccordancewimanembodimentwhereinmteractioninformanon 

used to deform representations of digital content prior to transmission to a remote device. 
[0028] Figure 12 is a system in accordance with an embodiment wherein user interaction with a 

representation of content on a first device can be used to improve a representation of the content on a shared 
device. 

[0029] Figure 13a is a first representation of a digital image having determined interaction areas 

in accordance with an embodiment of the present invention. 

[0030] Figure 13b is a second representation of the digital image of Figure 13a having a layout 

deformed using interaction information in accordance with an embodiment of the present invention. 



IS 
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DETAILED DESCRIPTION 
[0031] The invention is illustrated by way of example and not by way of limitation in the figures of 

the accompanying drawings in which like references indicate similar elements. It should be noted that 
references to "an" or "one" embodiment in this disclosure are not necessarily to the same embodiment, and 
such references mean at least one. 

[0032] In the following description, various aspects of the present invention will be described. 

However, it will be apparent to those skilled in the art that the present invention may be practiced with only 
all aspects of the present invention. For purposes of explanation, specific numbers, materials, and 
figurations are set forth in order to provide a thorough understanding of the present invention. However, 
it will be apparent to one skilled in the art that the present invention may be practiced without the specific 
details. In other instances, well-known features are omitted or simplified in order notto obscure the present 
invention. 

[0033] Parts of the description will be presented in data processing terms, such as data, selection, 

retrieval, generation, and so forth, consistent with the manner commonly employed by those skilled in the art 
to convey the substance of their work to others skilled in the art. As well understood by those skilled in the 
art, these quantities take the form of electrical, magnetic, or optical signals capable of being stored, 
transferred, combined, and otherwise manipulated through electrical, optical, and/or biological components 
of a processor and its subsystems. 

[0034] Various operations will be described as multiple discrete steps in turn, in a manner that is 

most helpful in understanding the present invention, however, the order of description should not be construed 
as to imply that these operations are necessarily order dependent. 
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[0035] Various embodiments will be illustrated in terms of exemplary classes and/or objects in an 

object-orientedprogrammingparadigm.Itwillte^^ 

canbepracticed using any number ofdifferent classes/objects, not merely those included here for illustrative 
purposes. Furthermore, it will also be apparent that the present invention is not limited to any particular, 
software programming language or programming paradigm. 

[0036] Systems and methods in accordance with embodiments of the present invention can provide 

for improved presentation and interaction with digital content and representations of digital content. 
Representationasusedherein includes, but is not limited to any visual and/or audible presentation of digital 
content. By way of a non-limiting example, digital images, web pages, digital documents, digital audio, and 
other suitable content can have corresponding representations of their underlying content. Moreover, 
interfaces such as graphical user interfaces (GUI's) can have corresponding representations of their 
underlying content. 

[0037] In one embodiment of the present invention, past and current user interaction with a first 

representation of digital content can be used to improve presentation and interaction with a second 
representation of the same digital content or of other digital content. For example, user interaction with a 
representation of an interface or web page can be used to improve the presentation and interaction with a 
scaled representation of the interface or web page. In one embodiment, user interaction can be used to 
deform the layout and/or active areas of a representation of digital content. User interaction with a 
representationsuchasthatofan interface or digital document can also be used to improve presentation and 
interaction with the same representation. In other embodiments, user interaction with a representation of first 
digital content such as a first web page can be used to improve a representation of second digital content such 
as a second web page. 
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[0038] Figure 1 is a flowchart for using user behavior while interacting with a first representation 

to deform a second representation in accordance with one embodiment of the present invention. Although this 
figure depicts functional steps in a particular order for purposes of illustration, the process is not limited to any 
particular order or arrangement of steps. One skilled in the art will appreciate that the various steps portrayed 
in this figure could be omitted, rearranged, combined and/or adapted in various ways. 
[0039] In one embodiment, the first representation and the second representation can be the same 

representation. In another embodiment, the second representation can be a scaled version of the first 
representation. In yet another embodiment, the first representation can be of first digital content while the 
second representation can be of second digital content. 

[0040] At step 102, user behavior while interacting with a first representation can be monitored. 

Monitoring user behavior while interacting with the first representation can include determining areas of the 
representation with which one or multiple users interact. Determining areas of the representation with which 
a user interacts, such as by providing input at an area or indicating a selection of an area, can include 
determining areas corresponding to input received from a user. For example, an area of the representation 
that a user clicks or selects can be determined. A selected hotspot, a highlighted textual passage, a selected 
menu item or button can all be determined as part of step 102. 

[0041] Any device or method for providing input can be used in accordance with embodiments of 

the present invention. For example, monitoring user behavior while interacting can include monitoring mouse 
clicks, keyboard selections, voice commands, gestures, remote control inputs, PDA inputs as well as any other 
suitable input. An application or software component such as a monitoring component canbe configured to 
accept information relating to input. By way of a non-limiting example, a component can be configured to be 
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notified of input such as mouse clicks. An event or an object generated by the input can be accessed to 
determine information such as the location and type of the input. 

[0042] Monitoringuserbehaviorcanincludemonitoringasingleuser'sbehaviorormultipleusers' 
behaviors. User behavior can also be monitored over periods of time. For example, user interaction with a 
representation over multiple instances or periods of time can be observed. The information can be used to 
determine user behavior based upon the observed extended interaction period or instances. In one 
embodiment, the user behavior of a firstuser while interactingwitharepresentationof digital contentcanbe 
used to deform a representation of digital content presented to a second user. 

[0043] Interaction information can be determined using the user behavior at step 104. In one 

embodiment, determining interaction information includes determining identific 

interacted with or detemined to be ofinterest to a user. For example, the pixel coordinates of a point or area 
of interaction can be determined, an identifier for a menu item of an interface can be determined, or a 
destinationuniversal resource locator (URL) corresponding to a selected hyperlink can be determined as part 
of determining interaction information. Furthermore, information identifying degrees or sequences of 
interaction with areas or sets of areas of the first representation can be determined. In one embodiment, 
determininginteracnon^^ 

interactions. Furthermore, determining interaction information can include aggregating, averaging, or 
otherwise combining multiple instances of a single user's interaction with a first representation. 
l0 044] At step 106, interaction information can be maintained. In one embodiment, maintaining 

interaction information can include logging or storing information relating to interaction. For example, 
informationidentifying areas interacted with or determined to be ofinterest using observed user interaction 
can be stored. As discussed below, numerous systems and techniques can be used to maintain interaction 



information. 
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[0045] A second representation can be deformed at step 108 based on tbe behavior monitored at 

step 102. Various techniques for deforming the second representation can be used in accordance with 
embodiments of the present invention as discussed below. 

[0046] Figure 2 is a system 200 in accordance with an embodiment that can be used to implement 

the flowchart of Figure 1. Although this diagram depicts objects/processes as logically separate, such 
depiction is merely for illustrative purposes. It will be apparent to those skilled in the art that the 
objects/processes portrayed in this figure can be arbitrarily combined or divided into separate software, 
firmware or hardware components. Furthermore, it will also be apparent to those skilled in the art thatsuch 
objects/processes, regardless of how they are combined or divided, can execute on the same computing 
device or can be distributed among different computing devices connected by one or more networks. 
[0047] Monitoring component 202 can monitor user behavior while interacting with one or more 

representations of digital content. As part of monitoring user interaction with a first representation, 
component202canreceiveinputcorrespondingtoareasoftherepresentationwithwhichauser'sinteract S . 

Forexample,monitoringcomponent202canreceiveinputidentifyinganareaofthe representation that a user 
selectsortowhichauserotherwiseprovides input. Such input can include "highlighting" a textual passage 
of content, selecting a portion of an image or document (e.g., "clicking and dragging" with an input device 
toidennfyorselectanarea),andsele^^ 

In some embodiments, monitoring component 202 can monitor how often or how many times an area(s) is 
interacted with or in what order a user interacts with areas. 

[0048] Inoneembodiment,monitoringcomponent202canmonitortheselectionofactiveareasof 
a representation. Active areas can include any geographic region of a representation of content adapted to 
receive user input and/or provide an action in response to user input. By way of a non-limiting example, 
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hyperlinks, hotspots, areas for receiving data such as textual data supplied by a user, and areas that otherwise 
have actions associated with selection of the area all have corresponding active areas. Menu items or buttons 
of an interface or of content such as web pages, documents, etc. can have corresponding active areas. By 
way of a further non-limiting example, a representation of a GUI for a software application may have an 
active area corresponding to menu items, buttons, data entry fields, etc. Menu items of an interface can 
include any area that can be selected to invoke, edit, or utilize the interface and the underlying application 
including its features, functions, attributes, options, or commands. 

[0049] Various techniques can be used in accordance with embodiments to monitor user behavior 

while interacting with representations of digital content. It will be appreciated by one of ordinary skill in the 
relevant arts that any of such techniques can be used and are within the scope of the present disclosure. By 
wayofanon-limitingexample,monitoringcomponent202canmonitorallorsomeinputreceivedbyadevice 

or application. Monitoring can include determining that input has been provided or received as well as 
determining the type of input. In one embodiment, monitoring component 202 can access a device driver or 
application programming interface (API). For example, an input device event such as a mouse click can be 
determined by accessing a device driver or API. Monitoring component 202 can monitor such events to 
determine that an interaction has occurred as well as what type of interaction has occurred and where the 
interaction has occurred. In one embodiment, component 202 is configured to be notified of input such as 
events. 

[0050] Logic component 204 can determine information and perform various functions and/or steps 

in accordance with the flowchart of Figure 1 . For example, logic component 204 can be used to determine 
interaction information using user behavior at step 104 of Figure 1. Logic component 204 can further 
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determine information andperform various functions and/or steps not shown in Figure 1. Logic component 
204 can determine interaction areas from the user behavior monitored by component 202 in one embodiment. 
[0051] Aninteractionareacanincludeanygeographicregionofarepresentationwithwhichauser 
interacts. Input received from a user can be used to determine and/or define an interaction area. In one 
embodimentforexample,monitoring component determines thatauser has provided input to a representation. 
Logiccomponent204 can accept the information and determine an area corresponding to the received input. 
By way of a non-limiting example, logic component 204 can receive or retrieve an event generated by an 
input device driver and determine a region of a representation corresponding to the event. 
[0052] Inasimplenon-limitingexample,logiccomponent204candetermineapixelcoordinate(s) 
correspondingtoaregion towhichauser has provided input. Information identifying an interaction area can 
include anx,ycoordinate corresponding to an interaction area. In other examples, two x,y coordinates such 
as twopointsidentifyingtwocornersofarectanglecanbe used to identify an interaction area or a coordinate 
and radius can be used to identify a ci^^ 

ofarepresentation. An area defined by the user's input can be determined and identified using coordinates 
corresponding to the received input. Any scheme or technique known in the art for identifying an area can 
be used in accordance with embodiments of the present invention. 

[0053] In one embodiment, an API of a software application can be accessed to determine an 

interaction area. For example, in many slide show presentation software applications, textual, image, and 
other areas in content presented through the interface are defined by boundaries such as text box boundaries. 
By accessing an API, a selection within these boundaries can be determined. From the selection, a textual 
or other region corresponding to the selection can be determined to be an interaction area. 
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[0054] inoneembodiine^ 

area. For example, logic component can receive an event indicating a single click within a textual passage. 
The singleclick can correspond to and be represented by a single pixel coordinate within the text passage. 
Logic component 204 can receive the event and use the information to extrapolate and determine an 
interaction area corresponding to the entire textual passage. For example, logic component can determine 
two or more pixel coordinates defining a region corresponding to the textual passage. 
[0055] Addinonally.receivinginputatalocationnearanareacanbeusedtodeterminethatanarea 

isaninteractionarea. Thiscanbeuse^^^ 

overaperiodoftimearebeingmonitored. The user(s) may not select the exact same area even though their 
interest is all in me same area. In this embodiment, selections near an area can be used to determine that 

area is an interaction area. 

[0056] A ctiveareascanalsobedeterminedtobeinteractionareas.Inoneembodiment,userinput 
received at or near an active area can be used to determine that an active area is an interaction area. For 
example, user selection ofahotspot or hyperlink can be used to determine that the active area corresponding 
to the hyperlink or hotspot is an interaction area. 

[0057] Logiccomponent204canadditionallydetermineuniqueidentifierscorrespondingtomenu 
items, buttons, andother similar active areas ofan interface or other content with which a user interacts. In 
one embodiment for example, an API can be accessed to determine that an active area corresponding to a 
menu item has been selected. A unique identifier such as a variable name for the menu item can then be 
determined. A unique identifier can be tracked to determine a corresponding active area of a second 
representation. 
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[0058] Uniqueidentifierscanalsobeu^^ 

page are used to deform a representation ofa second web page. For example, a "submit" button or field for 
entryofausemameinarepresentationofaw^^ 

to be an interaction area. An identification of the interaction area can be determined by accessing an API 
for an internet browser to determine the area of the page selected. After determining the area selected, a 
document file (e.g., an html file) for the web page can be accessed to determine a unique identification for 
the area. For example, a text entry field of a web page may be represented by an identification such as 
'username.input' in an html document. The identification from the html document can be tracked. 
[0059] Logiccomponent204canmrtherdetermineadestinationuniversalresourcelocator(URL) 

toidentifyaniiiten^ 

be accessed to determine links selected by a user. By accessing the API, the destination URL address of 
ahyperlinkcanbedetermined. Inone embodiment, the URL canbe maintained with an identification of the 
web page containing the hyperlink. 

[0060] In one embodiment, logic component 204 can determine an image corresponding to an 

interaction area as part of determining interaction information. By way of a non-limiting example, an area 
corresponding to a "close" menu item ofa first representation of an interface may be determined to be an 
interaction area. The menu item may be represented by a graphic such as an 'X' as is show in Figure 9. 
Monitoring C omponent202candeterminethatinputhasbeenreceivedorprovidedatanarea corresponding 
to the menu item. Logic component can then determine a graphic corresponding to the menu item. Logic 
component 204 may access an API in order to determine the graphic or file containing information for the 
graphic. Images canbe tracked for later use in deforming a second representation. An image comparison of 
the tracked graphic can be made with the graphics of a second representation. A matching graphic can be 
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determined to be a corresponding interaction area and deformed accordingly. Image comparison can be 
applied to determine corresponding areas between any two images. For example, a person's face may be 
determined to be an interaction area. The face can be tracked and compared to determine a matching face 
in subsequent images. In this manner, interaction information with a first image can be used to deform a 
representation of a second image. 

[0061] It will be understood by those of ordinary skill in the art that other information identifying 

interaction areas canbe used in accordance with embodiments of the present invention. Additionally, other 
techniques can be used to identify some of the types of interaction areas discussed herein. For example, the 
active area corresponding to a hyperlink could be identified by pixel coordinates as has been discussed. 
Similarly, coordinates can be used to identify interaction areas corresponding to menu items, etc. 
[0062] In accordance with one embodiment, logic component 204 can include a degree evaluation 

component and a sequence evaluation component (not shown). The evaluation components can evaluate user 
behavior while interacting with interaction areas as well as evaluate the interaction areas themselves. In one 
embodiment, evaluation component 206 can evaluate interaction information in conjunction with determining 
interaction information at step 104 of Figure 1. These evaluation components can evaluate interaction to 
determine degrees of interaction with interaction areas and to determine a sequence of interaction with 
interaction areas. An aggregated, averaged, or otherwise combined interaction of multiple users or single 
users over multiple instances can be used to evaluate user interaction. 

[0063] In accordance with an embodiment, a degree evaluation component can evaluate how often 

or many times an interaction area is selected. The evaluation can be used at step 104 to determine 
interactioninformationidentifyingdegreesofinteractionwithinteractionareas. For example, the number 
of times one or more users select an interaction area can be determined from an evaluation of user 
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interaction. Information identifying the user interaction can be determined and stored along with an 
identification of the interaction area. In anofo 

areas can be determined from an evaluation of user interaction. The number of times an interaction area is 
selected in relation to omer interaction areas can be determmed. Information identifying a relative intemcti 
can be tracked and used in deformation. An evaluation of user interaction can also be used to determine that 
some areas should not be tracked at all. For example, an area selected once may be determined to not be 
of sufficient interest to warrant tracking. 

[0064] A sequence evaluation component can evaluate user interaction to determine a sequence by 

which one or more users interact with a representation. User interaction can be evaluated to determine a 
sequence of interaction with interaction areas in one embodiment. The evaluation can be used at step 104 
to determine interaction information identifying a sequence of interaction with interaction areas. Various 
information identifying a sequence of interaction can be used in accordance with various embodiments. For 
example, the order in which one or more users select interaction areas can be determined. In one 
embodiment, the interactions of multiple users can be aggregated, weighted, averaged, and/or otherwise 
combined to determine sequence information. Information identifying a sequence of interaction canbe very 
useful when deforming a representation of a second interface by applying an animation sequence to 
interaction areas. 

[0065] Information maintenance component 208 can maintain interaction information. In one 

embodiment, maintenance component maintains information determined at step 104 of Figure 1. In one 
embodiment, a file or storage unit can be used to maintain the information. Byway of non-limiting example, 
interaction information can be stored or maintained in a cache memory or a more permanent memory. 
Furthermore, interaction information can be stored or maintainedby a client, server, or proxy depending on 
a particular application and/or the applications desired functionality. 
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[0066] Information relating to user interaction can be stored for use in deforming representations 

of digital content. In a simple example, an identification of interaction areas of a first representation can be 
stored for later use in deforming a second representation. Additionally, information relating to a degree of 
interaction with an interaction area can be stored. For example, a number of times an area was interacted 
with or a percentage of interaction with an interaction area relative to other interaction areas of a 
representation canbe stored. Furthermore, information relating to a sequence ofinteraction with interaction 
areas can be stored. The information in the file can be accessed and used to deform a representation of 
digital content when presented. 

[0067] In some embodiments, maintenance component 206 can maintain information relating to how 

arepresentationofan interface or other digital contentis to be deformed. This deformation information can 
bemaintainedalongwithinteractioninformation. For example, an identification of an interaction area can 
be stored with information relating to how to deform the interaction area. The information may specify that 
an area is to be enlarged or zoomed, an active area expanded, an animation to be applied to the area, etc. 
[0068] inoneembodimentofmepresentmvennon,interactioninformationismaintainedalongwith 
an identification of the digital content to which the interaction information relates. For example, a file can 
include information identifymg an mterface, an identification ofinteraction areas of me inter^^^ 
relating to the degree ofinteraction with each interaction area, and information identifying a sequence of 
interaction with the interaction areas. 

[0069] Consider an example wherein interaction information with a web page is tracked. A file used 

to maintain the interaction information may include information identifying the web page (such as its URL 
address), information identifying interaction areas of the web page (such as the URL address or pixel 
coordinates corresponding to hyperlinks followed from the web page), information identifying the order in 
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which the interaction areas were selected or are most commonly selected, and information identifying the 
number of times each interaction area was interacted with. 

[0070] A file or other storage unit that is used to maintain interaction information can include 

information for more than one representation of digital content. For example, one file may include 
identifications of numerous interfaces and other content such as web pages and documents along with 
interaction information for each of the interfaces and content. 

[0071] In one embodiment, interaction information can be added to or embedded within other 

information for digital content. By way ofnon-limiting examples, interaction information can be appended 
to, combined with, or imbedded within a digital document file (such as a .txt file, e.g.), a web page file (such 
as an .html or jsp file), or a file for a software application. The interaction information can be accessed to 
deform a representation of the digital content when rendered. 

[0072] Deformation component 208 can be used to deform a second representation of digital 

content. In one embodiment, deformation component 208 can be used to deform a second representation 
of digital content at step 108 of Figure 1 . Deformation component 208 can deform representations of digital 
content through a dedicated interface as well as deform representations of digital content rendered or to be 
rendered by other applications. For example, a representation of a digital document presented through a GUI 
of a word processing application can be deformed. 

[0073] It will be understood that numerous software applications and content presented with such 

applications can benefit from various embodiments. For example, and without limitation, internet browsers, 
mediaplayers/editors,wordprocessmgapplications,slidepresentation/creationapplications,e-bookreaders, 

and operating system user interfaces can all benefit from embodiments. In one embodiment, software can 
be incorporated within any such application. It will be understood that one or more components can be added 
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to an application. For example, an application may include a deformation component adapted to receive 
interactioninformation.Forexample, code maybeadded to the underlying code forthe application to improve 
the application. In other embodiments, a separate application can run "alongside" other applications for 
improvement of the application. 

[0074] In one embodiment for example, an application in accordance with an embodiment is 

implemented as a stand-alone application. A stand alone application may be implemented with a dedicated 
user interface for presenting, editing, and/or deforming representations such as web pages, documents, slides, 
etc. 

[0075] inanomerembodiment,abackgroundapplicationc^ 

and interaction with representations of web pages, documents, images, etc. presented through third-party 
interfaces as well as the presentation and interaction with third-party interfaces themselves. A background 
application can monitor user behavior while interacting with multiple representations including those of 
interfaces and other digital contentto improve presentation and interaction with other representations of the 
same or other digital content. 

[0076] In a background application implementation, user behavior while interacting with various 

content and applications can seamlessly be tracked to improve the presentation of software application 
interfaces and content presented through the interfaces. An application can monitor behavior, maintain 
interaction information, and apply deformations to representations of content and interfaces without user 
intervention and without interfering with the user's typical interaction with the interface or content. 
[0077] Embodiments in accordance with the present invention can be implemented as client side 

applications, server side applications, proxy applications, orvarious combinations thereof. Accordingly, the 
components of Figure 2 and steps of Figure 1 can be distributed and/or implemented in any combination 
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on a client-side device, a server, or a proxy server depending on the desired functionality of a particular 
application. All such combinations are considered within the scope of the present disclosure. 
[0078] For example, user behavior while interacting with a first representation can be monitoredby 

a monitoring component installed at a server, client, or proxy. Likewise, interaction information can be 
determinedbyalogic component and maintained by a maintenance component that is installed ataserver, 
client, or proxy. Moreover, a second representation can be deformed by a deformation component 
implemented at a server, client, or proxy. Additionally, in some embodiments, instances or copies of a 
component can be implemented at multiple locations. 

[0079] In a simple embodiment for example, monitoring component 202 can be implemented on a 

client device to monitor user behavior while interacting with representations of interfaces and content 
presentedontheclient device. Logic component 204 can determine interaction information and maintenance 

component 206 can track the information at the client device. If a second representation is to be deformed 
forpresentationontheclientdeviceusingtheinteractioninformation,adeformationcomponent208onthe 

client device can deform the second representation. If a second representation is to be deformed for 
presentation onasecond device usingthe interaction information, the interaction information can be accepted 
atthesecond device where a deformation component 208 can deform the second representation. In various 
embodiments, the deformation component can deform a representation using various techniques. A 
deformation component on a client device may access a document object model to deform corresponding 
interaction areas without modifying the original content. 

[0080] Other embodiments in accordance with the present invention can be implemented as server- 

side applications. User interaction with multiple devices in communication with the server can be used to 
determine interaction information. The information can be tracked at the server so that it can be used to 
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deform representations presented on any machine in communication with the server. In one embodiment, a 

server-side deformation component can modify original content before it is transmitted. 

[0081] Further embodiments can be implemented as proxy server based applications. A proxy 

server can interface between a client and server to handle transfers of information including content, events, 

commands, instructions, etc. For example, in a web-based context, a proxy server can handle requests and 

transfers among client based applications such as Web browsers and server side applications such as Web 

servers. 

[0082] In a proxy server based embodiment, interaction with representations of interfaces and 

content by one or more users can be maintained and used to deform representations of content passed 
through the proxy server. In one proxy based embodiment, monitoring component 202 can be implemented 
at a client (such as on a PDA, PC, etc.). Monitoring component 202 can monitor user interaction with 
representations of content presented on the client device. Logic component 204 can be implemented at the 
clientandcandetermineinteractioninformationbaseduponthemonitoreduserinteraction. The interaction 
information can be transferred to the proxy server using any system or technique as known in the art. 
[0083] In another embodiment, monitoring component 202 and logic component 204 can be 

implemented at the proxy server. By way of a non-limiting example, an object corresponding to a selection 
within a representation of a Web page containing Java script can be transferred to and/or through the proxy 
server. Monitoring component 202 can further monitor the selection of hyperlinks and other active areas 
within the representation. Monitoring component 202 can monitor messages including the transmission of 
objects and events corresponding to selections in order to monitor user interaction. Logic component 204 can 
; the received information to determine interaction information such as area corresponding to the selection. 
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Invariousproxybasedembodiments.monitoringandlogiccomponentscan 
proxy. 

[0084] Information maintenance component 206 can be implemented at the proxy server. In one 

embodiment, the maintenance component can be implemented as one or more files containing interaction 
information. In further embodiments, the maintenance can be maintained at any location including at a third 
party. Deformation and logic components can accept and/or retrieve information from the maintenance 
componentatthethirdparty.Inoneembodiment,aproxyorthird party implementation of the information 
maintenancecomponentcanbeusedtodeformarepresentanonofcontentforaseconduserusinginteracti^^ 
informationofafirstuser. Forexample,interactioninformationofafirstusercanbemaintainedataproxy, 
server, or on a network. A representation of content accessed by a second user through the proxy, server, 
or network can be deformed using the interaction information of the first user. 

[0085] inoneembodiment,deformationcomponent208isimplementedattheproxy.Attheproxy 
server, interaction information can be used in the deformation of representations of interfaces and content 
accessed through the proxy server. For example, a user may request a web page using a web browser or 
otherclientbased application. The proxy can retrieve the web page from a Web server (or locally if cached, 
e.g.). Deformationcomponent208 can use interaction information maintained by maintenance component 

206 to deform a representation of the requested web page before it is sent to the user. In one embodiment, 
adatafilesuchasan-htmldocumentand/orJavascriptcodecorrespondingtothewebpagecanbemodified 

priortosendingthefiletotheuser. Deformation component 208 can use previously determined interaction 
information with the requested web page or with other web pages or similar content. In one embodiment, a 
proxy based application can deform a representation of an interface or content based on a type of device 
receiving the content. 
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[0086] In another embodiment, deformation component 208 can be implemented at the client 

device. Interaction information can be transferred to the requesting client along with data for the requested 
web page. The deformation component can deform a representation of the web page before presentation on 

the client device. 

[0087] Figure 3 is flowchart in accordance with one embodiment for deforming representations of 

digital content. Although this figure depicts functional steps in a particular order for purposes of illustration, 
foe process is notlimited to anyparricu^ 

that the various steps portrayed in this figure could be omitted, rearranged, combined and/or adapted in various 
ways. 

[0088] In one embodiment, the flowchart can be performed in conjunction with step 108 of Figure 

1 when deforming a second representation of digital content. Figure 4 illustrates an exemplary system that 
can be used to deform representations of digital content. Although this diagram depicts objects/processes as 
logically separate, such depiction is merely for illustrative purposes. It will be apparent to those skilled in the 
art that the objects/processes portrayed in this figure can be arbitrarily combined or divided into separate 
software, firmware or hardware components. Furthermore, it will also be apparent to those skilled in the art 
that such objects/processes, regardless of how they are combined or divided, can execute on the same 
computing device or can be distributed among different computing devices connected by one or more 
networks. In various embodiments, the system of Figure 4 can be implemented as deformation component 
208 and/or to execute the flowchart of Figure 3. 

[0089] At step 302, interaction information can be accepted. In one embodiment, interaction 

information is accepted by receiving the information at deformation component 208 from information 
maintenance component 206. Information acceptance component 402 can receive and/or retrieve interaction 
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information from any storage location including from other applications. By way ofnon-limiting example, the 
acceptance componentcan access interaction information from dedicated storage units as well as from files 
containing information for digital content. 

[0090] At step 304, areas of a second representation corresponding to interaction areas of a first 

representationcanbedetermined. Area determiner 406 of logic components can determine corresponding 
areas in one embodiment. 

[0091] In accordance with the various types of information identifying interaction areas, area 

determiner 406 can determine corresponding interaction areas. Pixel coordinate(s) identifying an interaction 
area of the first representation can be used to determine a corresponding area of a second representation in 
oneembodiment.Correspondinginteractionareasofthesecondrepresentationcanbedeterminedby scaling 

the coordinate(s) in accordance with the ratio of the scales of the two representations. 
[0092] Unique identifiers can also be used to identify corresponding interaction areas of a 

representation. Corresponding interaction areas of a second representation canbe determined by identifying 
a corresponding unique identifier of a second representation. Various software applications may use the same 
or similar unique identifiers that can be used to deform corresponding areas of a second interface. For 
example, an interaction area corresponding to a menu item of a software application interface can be 
identifiedbyauniqueidentifier. A corresponding unique identifier ofa second application can be identified 
to determine a corresponding area in a representation of an interface of the second application. Unique 
identifiers can be used in embodiments where interactions with a first representation ofa GUI for a software 
application are used to deform a second representation of the GUI for the software application. Moreover, 
unique identifiers can be used where interactions with a GUI of a first software application are used to 
deform a GUI ofa second software application. 
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[0 093] Unique identifiers can also be used when interactions with a representation ofa first web 

page or similar document are used to deform a representation of a second web page or document. A 
document or file, such as an html document, for a second web page can be searched to identify a 
correspondingunique identified Destination 
URL addresses can be used to determine corresponding areas ofa second interface in applications for web 
pages. An interaction area of a first web page corresponding to a hyperlink can be identified using a 
destination URL address of the hyperlink. A file or document representing a second web page can be 
searched to identify a hyperlink having the same destination URL address. From the URL address, a 
corresponding area of a representation of the second web page can be determined. 
[0094] Image comparison can also be used to determine corresponding areas of a second 

representation. An image or other graphic corresponding to an interaction area can be used to identify an 
interaction area ofa first representation such as an interface. Area determiner 406 can compare the image 
or graphic with images or graphics of areas ofa second representation such as that of a second interface. 
A corresponding area of the second representation can be determined from a matching comparison. 
[0095] At step 306, a type of deformation to apply to the corresponding areas of a second 

representationcanbedeterm^^ 

can be maintained as part of the interaction information. A type of deformation can be stored along with 
interactioninformationde™^ 

determine the type of deformation to apply from the specified information. 

[0096] lnotherembodiments,atypeofdeformationisnotmaintainedwithinteractioninfo^^ 

In such embodiments, type determiner 408 can determine a type of deformation from a set of default types 
of deformation, user-specified type of deformation, and/or type determiner can determine a type of 
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deformation by evaluating the interaction information. In one embodiment, type determiner 408 includes a 
setofrules to be used in determining types of deformation. An interaction area can be evaluated and a type 
ofdeformationdeterminedusingtherules. For example, type determiner 408 may determine that active area 
deformation is tobe applied to corresponding active interaction areas and a zoomed layout deformation applied 
to other corresponding interaction areas using a set of rules. 

[0097] At step 308, an amount of deformation to apply to corresponding areas of a second 

representation can be determined. As with the type of deformation, information identifying an amount of 
deformationto apply to an interaction area can be maintained along with the interaction information in one 
embodiment. In other embodiments, an amount of deformation canbedeterminedbyamountdeterminer 410 
usingadefaultsetofdeformationamounts, user-specified amountsof deformation, and/or from an evaluation 
of the interaction information. A set of rules that can be configured by a user can be used in one 
embodiment. By way of a non-limiting example, amount determiner 410 can access information relating to 
degree and sequence of interaction with interaction areas to determine an amount of deformation. The 
amount of deformation applied to a corresponding interaction area canbe increased according to the number 
of times an area is interacted with or according to a relative interaction with an area. 
[0098] In one embodiment, determining a type and amount of deformation for areas of a second 

representation includes determining an animation sequence to apply to corresponding interaction areas. Logic 
component 404 can determine an animation sequence using degree and sequence of interaction information. 
For example, an order of deformation of interaction areas canbe determined from the sequence information 
and areas of greater interest or interaction canbe presented for longer periods of time during the animation 
rdeformedbyalargeramountduringthe animation. As with other deformations, animation sequences can 
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be maintained along with interaction information or animation sequences can be determined by logic 
component 404 from such information as the type of interaction area. 

[0099] In one embodiment in accordance with the present invention, types and amounts of 

deformation can be determined from user specified parameters. For example, an exemplary application of 
mepresentinventioncanin^^^ 

types of deformation. A user can select types of deformation, amounts of deformation, whether to apply 
deformation, and to what types of areas to apply deformations. A user can select whether or not to apply 
deformations and if so, what type of deformations to apply. A user can also select to have different 
deformations apply to different types of identified interaction areas. Additionally, a user can select to have 
deformations applied to representations of some application interfaces and content and not to other 
representations of interfaces and content. For example, in a background application implementation, a user 
could select to only have deformations applied to her internet browser or word processing program and 
contentpresented through the interfaces ofthe programs. Furthermore, a user can elect to have deformations 
applied tocertainrepresentationsofdigitalcontentandnototherrepresentationsofthedigital content. For 
example,ausermayselecttohave deformations applied to representations ofsoftware application interfaces 
when viewing them on a PDA and not when viewing them on a laptop or desktop. In one embodiment, the 
user specified parameters can be used to determine an amount and type of deformation at the time of 
rendering a second representation. In other embodiments, the user specified parameters can be maintained 
along with interaction information. 

[0100] AtstepSlO.theamountandtypeofdeformationcanbeappliedtoasecondrepresentation. 
In one embodiment, deformation application component 412 can apply deformation to a second 
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representation. Using the information determined at steps 306 and 308, the deformation can be applied to 
the corresponding areas determined at step 304. 

[0101] By way of non-limiting example, deformation application component 412 can apply 

deformation to a second representation by accessing a document object model ofdigital content in order to 
change the content's appearance or representation. A specification of how objects are represented in the 
content can be determined. Furthermore, how attributes are associated with objects and how each can be 
manipulated can be determined. The attributes and objects can then be manipulated in accordance with 
interaction information. In one embodiment, the original digital content is not modified in order to deform the 
secondrepresentation. mother embodiments, digital content itselfcan be modified and/or manipulated. In 
one embodiment, the deformation can be applied dynamically as a representation is rendered. 
[0102] In one embodiment, deformation is only applied to representations that are scaled. A 

representation size may be compared to a nominal representation size of the same or similar digital content. 
Iftherepresentationisdetermined to be scaled representation, interaction information can be used to deform 
the representation. In another embodiment, deformation is only applied to representations that are scaled- 
down. 

[0103] Deformation can be applied to an original representation of digital content and/or other 

representations of the same digital content or of different digital content. For example, a user may interact 
witharepresentationof digital content on a desktop machine and another representation of the digital content 
on a PDA. User interaction with representations presented on the desktop machine can be used to determine 
interaction information that will be used to deform representations of the same content on the PDA. 
Deformation can be applied to a representation of the content on the PDA without the deformation 
propagatingtotheoriginal representation orto the underlying digital content itself, although the deformation 
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canpropagatetolheoriginalrepresentationorunderlyingcontentifsod^ 

a representation on the desktop can be used to deform the same representation of the same digital content 
on the desktop, a different representation of the same digital content on the desktop, or a representation of 
different digital content on the desktop. It will be apparent to those of ordinary skill in the art that various 
implementations can be utilized depending on the needs of a particular application in accordance with 
embodiments. 

[0104] Various types of deformation can be applied to representations of digital content in 

accordance with various embodiments. As previously discussed, active areas can include any area capable 
ofre<*ivingusermptf 

by a graphical representation of the active area. In many cases, active areas correspond directly with a 
graphical representation of the active area. For example, an active area corresponding to a hyperlink often 
covers the same area as a graphical representation of the hyperlink (e.g., a text passage, image, etc.). 
However,anactive area canbeexpanded or contracted otherwise made to be ofa size not commensurate 
with the graphical representation of the active area. 

[0105] Figure 5a illustrates an exemplary first representation 502 of digital content. By way ofa 

non-hmitingexample, first representation 502 could be of an interface such as a word processing program 
orofsomecontentsuchasawebpage, document, image, or ofsome other suitable content. Representation 
502 can include active areas as shown by the boxed lines. If representation 502 is of an interface, the active 
areas may correspond to menu items, etc. If representation 502 is of a web page, e.g., the active areas may 
correspond to hyperlinks, text input fields, or other selectable areas. The boxed lines illustrate the active area 
itself and not the graphical representation of the active area. As illustrated, active areas 508a - 508d have 
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been determined to be interaction areas. The underlining is provided merely for purposes of explanation and 

is not intended to represent actual content that would be presented in the representation. 

[0106] FigureSbillustratesanexemplarycorrespondingsecondrepresentationSSl.Asdiscussed, 

second representation 552 could be a scaled version of first representation 502, a representation of an 
interface for a second application, a representation of different digital content such as a second image, etc., 
orevenmesamerepresentationasfirst^^ 

first representation 502, corresponding interaction areas of representation 552 can be deformed. As 
illustrated, active interaction areas 558a-d have been expanded relative to the overall size of the 
representation. Thus, a user can more easily select the interaction areas while still being able to select 
adjacent areas as well. The larger selectable areas can facilitate eased navigation and selection. When 
deforming active areas, includingmenu items, etc., the graphical representation of the active area or menu 
item need not be modified, although it can. In one embodiment, only the active area itself is deformed. In 

otherenfcodi^ 
as discussed below. 

[0107] Active areas can be deformed using other techniques and methods in accordance with 

embodimentsofthepresentinvention. Figures 5a-5b illustrate a simple example ofexpanding the selectable 
region of an active area. In another embodiment, the selectable regions can be expanded such that they 
overlap adjacent active areas. In yet another embodiment, selectable regions of adjacent active areas can 
be contracted to facilitate selection of interaction areas. 

[0108] inoneembodimentinaccordancewiththepresentinvention,userinteractionandbehavior 
can be used to deform the layout of a representation. User behavior and interaction can be used to deform 
any area of a representation, not just that of active areas. Layout deformation differs from deforming an 
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active area itself in that the graphical representation of an area is deformed, rather than an active area. If 
a graphical representation of an area corresponds to an active area, the active area may or may not be 
deformed. It will be understood by those of ordinary skill in the art that various combinations and 
subcombinations of the deformation systems and methods discussed herein can be useful in various 
applications and systems. 

[0109] Deforming the layout of active areas corresponding to interaction areas can provide for 

improved interaction with an interface and/or other representations of digital content. Deformation of the 
layoutofactiveareascorrespondingto interaction areas can provide useful feedback to a user that they can 
interact atanaiea. The deform^^ 

an active area. Additionally, deforming the layout can improve the presentation of an interface or content 
and make the interface and content more engaging. 

[0110] Numerous deformation techniques can be used in accordance with embodiments of the 

present invention for layout deformation. For example, zooming in/out at an interaction area, presenting a 
fisheye perspective of interaction areas, using a tool tip feature to provide additional content such as what 
would be provided if an active area corresponding to an interaction area were selected can all be used. 
Deformation can be combined with other techniques to further enhance the presentation of an interface and 
content. Inone embodiment, the layout ofinteraction areas can be deformed based upon input received from 
a user. Deformation can occur upon a user placing a cursor over an interaction area or upon a user placing 
acursorover an interaction area and simultaneously providing input by selecting with an input device such 
asmouse. Furthermore, in touch-screen based interactions such as with a PDA, a layout deformation of the 
interacnonareascanbegina^^ 

areas can be deformed as the cursor is moved across the area. By waving or moving a cursor across the 
interface, a user can quickly identify points of interest based upon past user interaction. 
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[0111] inoneembodimentofthepresentinvention,layoutdeformation C anbeimplernentedasan 

ammation.Asillustoted^ 
6aillu S ttatesafirstrepresentanon600ofdi^^ 

606 have been determined to be interaction areas. The boxed lines around the interaction areas have been 
provided merely for purposes of explanation. They are not intended to represent actual content of the 
representations. Figures 6b - 6d illustrate a second representation 610 of the digital slide over a period of 
time. An animation isapplied to the representation such that corresponding interaction areas 602', 604', and 
606' are each enlarged for a period of time to bring the user's attention to the interaction areas. In various 
embodiments, the animation can be presented once when representation 610 is rendered or can repeat 
multiple times. 

[0112] lnoneembodiment,anaiumationcanbeimplementedtooccurautomaticallyuponrendering 
a representation. In other embodiments, user input can be combined with animation to quickly present 
interaction areas to a user. For example, passing a cursor over a representation can trigger an animation of 
interactional. The animationcan^^ 

vertical position ofthe cursor is used topresentananimationofunderlyinginteractionareas. Bypassingthe 
cursor up or down the representation, all interaction areas at a similar vertical location to the cursor can be 
deformed using an animation. The deformation can occur regardless of whether the cursor actually passes 
over the interaction area. 

[0113] Deformation can also be dependent on the nature of the tracked user interaction and 

behavior. In one embodiment, the number of times a user interacts with a particular area of a first 
representation can be tracked. When the interaction area is rendered in a second representation, the number 
oftimesthe interaction area was selected canbe used to determine the amount of deformation to apply. The 
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amountofzoomortowhatdegree an area is enlarged canbe determined using the number oftimes the user 
interacted. 

[0114] In one embodiment, the number oftimes an interaction area is selected relative to other 

interaction areas can be used to determine an amount of deformation. In addition, the type of deformation 
can depend upon the degree of interest determined for an area. For example, frequently selected areas can 
be automatically enlarged or animated while less selected areas can be enlarged upon cursor over. 
[0115] In one embodiment in accordance with the present invention, a sequence of interaction or 

behavior with a first representation can be used in deforming the layout of a second representation. By way 
ofanon-limittegex^ 

of the data provided in the box. As illustrated in Figure 7, a representation 702 of a web page on a first 
device 700 can include text box 704 to receiveuser data and a "submit" button 706 to instigate transmission 
ofthe data to the webpage provider. It can be determined by observing and tracking user behavior with the 
representation of the web page that one or more users interacting with the page typically provide text in the 
box, then position a cursor over the submit button, and then select the button. The tracked information 
regardingthesequenceofinteractioncanbeusedwhenasecondrepresentationofthewebpage is rendered. 

[0116] As illustrated in Figures 7b and 7c, a scaled representation 702' ofthe web page has been 

renderedonPDA708. Figures 7b and 7c illustrate the representation over some period of time. Information 
regarding interaction with representation 702 can be used to create an animation whereby text box 704' is 
first enlarged to facilitate receiving data as illustrated in Figure 7b. Thereafter, as illustrated in Figure 7c, 
submitbutton 706' canbe enlarged to facilitate selection thereof. In one embodiment, the animation is time 
based, deforming interaction areas for periods of time. The animation can also be based on user input. A 
first interaction area can be deformed based upon the determined sequence of interaction. After receiving 
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input, a second interaction area can be deformed. The above animation technique can also be combined with 
active area deformation to further improve the usability of the interface and content. In one embodiment, a 
first user's interaction with a representation of the web page can be used to determine interaction information 
that can then be used to deform a representation of the web page presented to a second user. 
[0117] In various embodiments, any interaction can be deformed or only interaction areas 

corresponding to active areas can be deformed. In some applications, it can be useful to deform all interaction 
areas and then deform only interaction areas corresponding to active areas. Additionally, a first type of 
deformation may be applied to interaction areas corresponding to active areas and a second deformation 
applied to all other interaction areas. For example, it may be useful to deform non-active interaction areas 
by enlarging the area upon cursor over. For active interaction areas, however, it may be useful to 
automatically expand them. Additionally, if a non-active interaction area is enlarged such that it overlays an 
active area, an indication such as a visual highlight around the non-active area can be used to indicate the 
presence of an active area below the non-active enlarged area. 

[01 181 As discussed above, systems and methods in accordance with embodiments of the present 

invention can be used to improve functionality in various software applications. For example, software is 
available that can allow an author to create a presentation such as a slide presentation and then present the 
presentation to a group of end-users, whereby the end-users can view the presentation on a personal device 
such as PDA, laptop, cell phone, etc. The devices of the presenter and the end-users may be connected to 
a network such that the presenter can control the presentation and the slides currently selected by the 
presenter can be presented to the end-users. Slide as used herein is intended to reference any content forming 
part of a digital presentation. By way of a non-limiting example, a slide can be or include audio, video, images, 
and any other suitable content. In one embodiment, a slide is captured and presented at an end-user's device. 
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[0119] Figure 8 is a system in accordance with an embodiment for using interaction information 

with a meeting software application. The interaction of presenter 804 with a representation of the 
presentation can be monitored while authoring and/or presenting the presentation. Interaction may be 
monitored on any device including computer 802. In an embodiment, a monitoring component can be 
implemented at network 806. 

[0120] Monitoring components on network 806 and/or devices 808 - 812 can monitor user 

interaction with representations of the presentation. Monitoring user interaction can include monitoring end- 
user selection of areas within a representation of the presentation slides and/or a representation of an 
interface of a meeting software application. 

[0121] Interaction areas can be determined from user interaction with representations of the 

presentation. A logic component such as logic component 204 can be implemented at network 806 or on 
devices 808 - 81 2 to determine interaction information. Interaction within areas (textual, image, etc .) defined 
byanauthorofthepresentationorbyapplication itself canbe used to determine that an area definedbythe 
boundaries is an interaction area. End-users can also select areas using an input device such as by 
"highlighting" an area. For example, an end-user may highlight a textual passage in order to "copy" it to 
another interface, file, application, etc. An area defined by any such interaction can be determined to be an 



interaction area. 



[0122] l n oneembodiment,ameetingsoftwarepresentationapplicationcanallowuserstoappend 
annotations to the content of a presentation. An annotation can include the addition of any content such as 
text to a presentation or portion thereof. Annotations can propagate to representations of the presentation 
on everyone's device or to a local representation on a device of the user who is interacting. 
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[0123] In one embodiment, pre-defined areas of content such as text boxes 820 and 822 shown in 

the expanded display of device 812 can be provided so that an end-user can select an area to provide 
annotations thereto. These annotatable areas can have corresponding active areas such as active area 826 
oftextbox820.Anareacanbedefinedbyanauthorofthepresentationorbyaprovideroftheapplication. 

An end-userviewingthe present*^ 

embodiment, the user's selection of such an area is used to makethe area active (selectable for annotation). 
Forexample,ausermayuseamouse to highlight a text passage such as passage 824 and add an annotation 
to the passage. In response to the user's selection of the passage, the passage can be made active on 
representations of the slide. An area to which a user has provided an annotation can also be determined to 
be an interaction area. 

[01 24] Interaction information can be maintained by a maintenance component such as maintenance 

component206atindividual devices or at network 806. For example, a file containing interaction information 
can be stored at network 806. Representations of the presentation on oneormore devices can be deformed 
usingthe maintained interaction information. The deformation can include deforming a representation on all 
devices or only on select devices. 

[0125] Adeformationcomponentcanbeimplementedatnetwork806and/ormeindividualdevices. 
In one embodiment, a deformation component at network 806 can use interaction information to deform 
representations of the presentation before they are provided to the individual devices. For example, before 
a slide captured for transmission to an individual device is transmitted to a device, data for the slide can be 
modified in accordance with the interaction information. 

[0126] In another embodiment, a deformation component or an individual device can accept 

interaction information and deform a representation before it is rendered on the device. A slide may be 
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provided to the device. Interaction information can be provided to the device as well. The information can 
be imbedded in data for the slide or can be provided in a separate file. The deformation component can apply 
the interaction information to the slide before it is rendered on the device. 

[0127] In one embodiment, interaction information used in conjunction with meeting software 

applications can be used to improve the presentation of a representation to a second user based on the 
interactions of a first user. For example, a first user may select an annotatable area or highlight an area. The 
interaction information determined from the first user can be used to deform a representation presented to 
a second user. The deformed representation can include deformed active areas that allow for easier 
selection, a deformed layout to enhance the presentation and/or bring a user's attention to areas of interest, 
and also newly defined active areas. The first user's interaction (e.g., highlighting a text passage) can be 
used to make a corresponding text passage of a representation presented to a second user an active area. 
Thus, the second user need only click within the new active area to select the text passage. The second user 
need not highlight the entire passage to select it. 

[0128] In one embodiment, a user can save one or more slides of the presentation locally. In one 

such embodiment, interaction information can be applied such that the saved representation is deformed. In 
another embodiment, interaction information can also be stored locally so that a representation of the 
presentation can be deformed thereafter when rendered. 

[0129] Virtual networks allow users to remotely access devices and content from any location. For 

example, a user may use a desktop computer connected to an office network during the workday. Using 
virtual network software, the user can remotely access his desktop machine, another device on the office 
network, applications on the network, and other content on the office network from anywhere. The user may 
use virtual network software on his PDA, cellphone, or laptop in order to remotely access his work network 
over the internet. 
Attorney Docket No. : FX/A20O5 

dcromer/fxpl/1061US0/1061us0.001.app.wpd "38- 



[0130] In many virtual network applications, a representation of content on a remote network is 

transmitted to the user's remote device. In some applications, the user receives an image representing 
content on the network. The user can interact with the representation of the image on the remote device. 
User input can be transmitted to the network where it can be processed by an application running on the 
network. 

[0131] Some virtual network applications scale the image for presentation on the end-user's device. 

For example, a representation of an internet browser interface may be scaled-down so that a user can view 
the entire interface. If the transmitted image is scaled-down, a user must interact with the smaller 
representation, thus making interaction more difficult. 

[0132] Figure 9 illustrates a system whereby a device 902 (a desktop computer as shown) 

connected with a network 904 is accessed using a remote device 910 (a PDA as shown). As shown, the 
devices are in communication with each other over networks 904, 906 and 908. It will be understood that 
the two devices can communicate in any suitable manner as is known and understood in the art. For example, 
both devices may be connected to a single network, both devices may be in communication over a dedicated 
connection, or both devices can communicate over a plurality of networks. 

[0133] A representation 922' of content presented on the PDA has been deformed using interaction 

information determined from user interaction with a device connected with network 904. Representation 922 
depicts content of desktop computer 902. However, no representation need actually be rendered by 
computer 902 or network 904 while a user is remotely accessingthe device or network.. Representation 922 
is shown for clarity of discussion. It should be noted thatthe user need not be controlling the same desktop 
computeronwhichinteractionwasmonitoredtodetermineinteractioninformation. Interaction information 
, be maintained on the network and accessed when a user is connected with the network in any way. 



can 
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[0134] Figure 1 0 is a flowchart in accordance with an embodiment for deforming representations 

of content on a remote device that is accessing a network and/or another device using a virtual network 
software application. Although this figure depicts functional steps in a particular order for purposes of 
illustration, the process is not limited to any particular order or arrangement of steps. One skilled in the art 
will appreciate that the various steps portrayed in this figure could be omitted, rearranged, combined and/or 
adapted in various ways. 

[01 35] At step 1002 of Figure 10, user interaction with representations of digital content on one 

or more devices (such as desktop computer 902) are monitored. A monitoring component can be 
implemented at device 902, network 904, network 906, and/or network 908. 

[0136] Interaction with representations of web pages, software application interfaces, and other 

content can be monitored. The interaction can be used to determine interaction information at step 804. 
Determining interaction information may include determining interaction areas of representations of multiple 
documents, web pages, and GUI's. Interaction area information may be evaluated to develop degree and 
sequence of interaction information. One or multiple users' interaction with one or multiple devices can be 
monitored to develop the interaction information. A logic component can be implemented at device 902, 
network 904, network 906, and/or network 908 to determine interaction information. 
[0137] At step 1006, the interaction information can be maintained. An information maintenance 

component at any device or network can maintain the information. The information may be persisted 
anywhere on the networks, including as a file on a storage device or on a computing device on which 
interaction was monitored. In one embodiment, the interaction information is stored at a proxy server. In one 
embodiment, a file containing interaction information of a user can be associated with the user using a 
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username, for example, so that when a user logs in to the network, his individual interaction information can 
be used to deform representations of content the user is accessing. 

[0138] At step 1008, a remote user is logged into the network using a standard virtual network, 

software application on a remote device such as PDA 910 for example. At step 1010, the virtual network 
application can transfer user input from the remote device to the network and on to a device connected with 
the network. At the network, the user input can be received and used to transfer a representation of content 
requested by the user to the remote device. For example, a representation of the "desktop" (including, e.g., 
an operating system interface and other application interfaces being used) of desktop computer 902 can be 
transferred to the user. Thus, the user can remotely control the desktop computer. 
[0139] Figurellaisaflowchartinaccorckncewimanembodimentwhereininteractioninformation 

is used at the PDA to deform representations of content. Although this figure depicts functional steps in a 
particular order for purposes of illustration, the process is not limited to any particular order or arrangement 
of steps. One skilled in the art will appreciate that the various steps portrayed in this figure could be omitted, 
rearranged, combined and/or adapted in various ways. 

[0140] In one embodiment, the flowchart of Figure 11a can be used as part of step 1010 of Figure 

10. At step 1 102, a representation(s) of digital content can be received by the remote device. As illustrated 
in Figure 9, a first representation 922 of an email application is presented on device 902. Since PDA 910 
is controlling desktop computer 910, a second representation 922' of the email application is presented on 
the PDA. As illustrated the second representation has been deformed. 

[0141] At step 1 104, interaction information can be received at PDA 910. PDA 910 can retrieve 

the interaction information or the information can be received along with the representation(s) at step 1 102. 
At step 1106, the interaction information can be used at the PDA to deform the representation(s) received 
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from the desktop device. For example, if an image representing content on the desktop is received, areas 
corresponding to interaction areas in the representation(s) could be enlarged or zoomed, etc. At step 1108, 
the deformed representation can be rendered on the remote device. 

[0142] Figure 1 IB is a flowchart in accordance with an embodiment wherein interaction 

information is used to deform representations of content prior to transmission to a remote device suchas PDA 
910. Although this figure depicts functional steps in aparticular order for purposes of illustration, the process 
is not limited to any particular order or arrangement of steps. One skilled in the art will appreciate that the 
various steps portrayed in this figure could be omitted, rearranged, combined and/or adapted in various ways. 
[0143] At step 1122, a device on the network such as desktop computer 902 receives a user 

request from PDA 910 for content on the network. At step 1124, interaction information is retrieved. At 
step 1126, the interaction information is used to deform a representation of the requested content. At step 
1128, the deformed representation of the requested content is transmitted to the remote device. 
[0144] Referring to Figure 9, areas 930, 932, and 934 of first representation 922 are determined 

interaction areas of a representation of the requested content. Accordingly, corresponding areas 930', 932', 
934' ofthe second representation rendered on display PDA910 have been deformed. Thus, user interaction 
with representations of content presented on desktop computer 902 and/or other devices on network 904 
have been used to deform the layout of representations of content presented on PDA 910. In this manner, 
a user's previous interaction with content can be re-used to enhance the usability of scaled-representations 
of the same content. 

[0145] Figure 12 is a system in accordance with an embodiment wherein user interaction with a 

representation of content presented on a personal device can be used to improve a representation ofthe 
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content presented on a shared device. Dynamic bulletin boards, one type of shared device, can be used in 
public areas, work environments, and other areas to share information with groups of people. 
[0146] Video display "bulletin board" 1200 can be used to share digital content and present 

representations of content to a plurality of people. By way of a non-limiting example, employees can post 
digital content to a bulletin board such that a representation of the content can be displayed on the bulletin 
board. Employees can post web pages, documents, presentations, or any other digital content found to be of 
interest. In many applications, multiple representations may be displayed on the board simultaneously. 
[0147] Representations 1202, 1204, and 1206 of web pages have been posted to bulletin board 

1202 are presented. Users connected with a network on which the board is connected can post content to 
the board as is known and understood in the art. For example, a server may host the content of the display 
and users can designate certain content using an application provided by the server. 
[0148] As can be seen, an individual observing the bulletin board is presented with a considerable 

amount of content. Depending on a user's location relative to the display, viewing and interacting with the 
representations may be more difficult in their scaled form. 

[0149] In one embodiment, a user's interaction with a representation of content is used to deform 

a representation of content presented on the board through the display. For example, an employee may 
interact with a web page at his desktop computer and then post the page to the board. The user' s interaction 
with a representation of the web page on his desktop can be monitored, interaction information determined, 
and the information tracked. When the user posts the page to the board, the interaction information can be 
used to deform a representation of the page when presented on the display. A deformation component in 
accordance with an embodiment on a network connected with the display can access the interaction 
information and deform the representation of the web page presented on the display. 
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[0150] Representation 1202 of a web page includes interaction areas 1210, 1212, 1214, 1216 and 

1218 that correspond to interaction areas determined by observing user interaction with another 
representation of the web page. In this example, corresponding interaction areas 1210, 1212, 1214, 1216, 
and 1218 of representation 1202 are enlarged. 

[0151] By deforming the representations using the tracked interaction information, a viewer of the 

board can be presented with more targeted content. The representation can identify to the user which areas 
were determined to be have been of interest to a previous user. Thus, a viewer of the board can more quickly 
and easily absorb and otherwise take in the content presented by the representation. Furthermore, active 
areas can be deformed to improve interaction with the representations. 

[01 52] In another exemplary embodiment of the present invention, interaction with a representation 

of a digital image can be used to deform a scaled representation of the image such as a thumbnail. A user 
may interact with a representation of a digital image in order to edit or manipulate the image. For example, 
a user may utilize a photo editing application having a user interface to select certain areas of an image for 
editing. A user may select an area for copying or to adjust the brightness, color, contrast, etc. of a selected 
area. A user may further select particular areas in order to leave an annotation on the area. 
[0153] interactionareaswimintheimagecanbedetenninedbymonitoringuserinteractionwiththe 

image. For example, an API of an editing application can be accessed to determine events such as mouse 
clicks within the image. A user may use an input device to select particular areas of the image for editing, 
copying, annotating, etc. Areas selected by a user can be determined to be interaction areas. The interaction 
areas can be tracked and used to deform a scaled representation of the image. In another embodiment, image 
comparison can be used. Thus, interaction with a first representation of a first image can be used to deform 
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a representation of a second image. For example, face recognition software can be used to identify 
corresponding areas such as the same face in a representation of a second image. 
[0154] Figure 13a illustrates an exemplary representation 1300 of a digital image without 

deformation. The representation could be a full size representation of the image presented on a desktop, etc. 
Areas 1302, 1304, 1306, 1308, and 1310 (dotted lines indicate the area determined to have been an 
interaction area and do not represent any part of the actual representation) have been determined to be 
interaction areas based on monitored user interaction with this representation of the digital image or another 
representation of the image. 

[0155] Figure 13b illustrates a second representation 1310 of the digital image having applied 

deformations. Second representation 1310 could be a thumbnail or other scaled representation of the digital 
image. As shown, the layout corresponding to interaction areas 1302, 1304, 1306, 1308, and 1310 has been 
deformed. The corresponding interaction areas (1302', 1304', 1306', 1308', 1310') have been enlarged. 
The enlarged areas can facilitate eased understanding of and interaction with the scaled representation of the 
image. Other deformation techniques can be applied in addition to or in place of the shown deformation. For 
example, the interaction areas could be animated such that each interaction area is enlarged in turn. Such 
an example can help users understand the image without having to interact with it. 
[0156] In one digital image example, a background application in accordance with an embodiment 

of the present invention can access interaction information to deform a representation of a digital image 
presented through a typical user interface such as those of internet browsers and photo editing software. In 
further embodiments, an application deforms a representation only when the representation is scaled in 
relation to a nominal or typical representation of the digital content. 
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[0157] One embodiment may be implemented using a conventional general purpose or a specialized 

digital computer or microprocessor(s) programmed according to the teachings of the present disclosure, as 
will be apparent to those skilled in the computer art. Appropriate software coding can readily be prepared 
by skilled programmers based on the teachings of the present disclosure, as will be apparent to those skilled 
in the software art. The invention may also be implemented by the preparation of integrated circuits or by 
interconnecting an appropriate network of conventional component circuits, as will be readily apparent to 
those skilled in the art. 

[0158] One embodiment includes a computer program product which is a storage medium (media) 

having instructions stored thereon/in which can be used to program a computer to perform any of the features 
presented herein. The storage medium can include, but is not limited to, any type of disk including floppy 
disks, optical discs, DVD, CD-ROMs, microdrive, and magneto-optical disks, ROMs, RAMs, EPROMs, 
EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems (including 
molecular ICs), or any type of media or device suitable for storing instructions and/or data. 
[0159] Stored on any one of the computer readable medium (media), the present invention includes 

software for controlling both the hardware of the general purpose/specialized computer or microprocessor, 
and for enabling the computer or microprocessor to interact with a human user of other mechanism utilizing 
the results of the present invention. Such software may include, but is not limited to, device drivers, operating 
systems, execution environments/containers, and user applications. 

[0160] The foregoing description of the preferred embodiments of the present invention has been 

provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the 
invention to the precise forms disclosed. Many modifications and variations will be apparent to the 
practitioner skilled in the art. Embodiments were chosen and described in order to best describe the principles 
of the invention and its practical application, thereby enabling others skilled in the art to understand the 
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invention, the various embodiments and with various modifications that are suited to the particular 
contemplated. It is intended that the scope of the invention be defined by the following claims and t 
equivalents. 
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